Skip to content

Operator: full reconciler implementation with status reporting and tests#34

Draft
Copilot wants to merge 3 commits intomainfrom
copilot/agentic-layer-testbench-27-reconciler-implementati
Draft

Operator: full reconciler implementation with status reporting and tests#34
Copilot wants to merge 3 commits intomainfrom
copilot/agentic-layer-testbench-27-reconciler-implementati

Conversation

Copy link
Contributor

Copilot AI commented Mar 17, 2026 •
edited
Loading

Implements the reconciliation loop for the Experiment CRD, replacing the scaffolded stub. Creating an Experiment CR now generates a ConfigMap, a Testkube TestWorkflow, and optionally a TestTrigger, all with ownerReferences for cascade deletion.

Reconciliation logic

  • ConfigMap — serializes inline scenarios to experiment.json (snake_case matching the Python testbench schema); empty-scenarios placeholder for dataset mode
  • TestWorkflow (testworkflows.testkube.io/v1, via unstructured.Unstructured)
    • Scenarios mode: mounts ConfigMap as /data/datasets/experiment.json; chains run → evaluate → publish → visualize
    • Dataset mode: prepends setup-template with resolved datasetUrl (HTTP or s3://bucket/key); no file mount
    • Always injects OTEL_EXPORTER_OTLP_ENDPOINT from otel-config ConfigMap
  • TestTrigger (tests.testkube.io/v1) — created when spec.trigger.enabled: true, watching the Deployment derived from agentRef; deleted when disabled/removed
  • Agent URL resolved as in-cluster DNS: http://<agentRef.name>.<agentRef.namespace>:8000
  • Graceful no-op when Testkube CRDs are absent (isCRDNotInstalled helper)

Status reporting

Sets Ready and WorkflowReady conditions (with ObservedGeneration) and populates status.generatedResources after each reconcile.

Tests

23 envtest unit tests covering scenarios mode, dataset mode (including S3 URL resolution), trigger create/delete lifecycle, status conditions, and idempotency. Minimal Testkube CRD fixtures added to testdata/crds/ so envtest can accept unstructured TestWorkflow and TestTrigger objects.

golangci-lint v2 migration

The existing .golangci.yml was broken against the installed golangci-lint v2.10.1. Updated to the v2 config schema: version: "2", exclusion rules moved to linters.exclusions.rules, removed deprecated linters (exportloopref, gofmt, goimports, gosimple, typecheck), and pinned go: "1.25" to match the linter binary's build toolchain.

Original prompt

This section details on the original issue you should resolve

<issue_title>Operator: Reconciler implementation, status reporting, and tests</issue_title>
<issue_description>## Parent Issue
Part of #27

Description

Implement the full reconciliation logic with all resource generation, status management, and test coverage.

Scope

ConfigMap generation

  • Serialize inline scenarios to experiment.json ConfigMap
  • Create placeholder ConfigMap for dataset mode (populated by setup phase)

TestWorkflow generation

  • Build TestWorkflow spec chaining phase templates (setup → run → evaluate → publish → visualize)
  • Conditionally include setup-template only when spec.dataset is set
  • Skip setup and inject experiment.json via ConfigMap when spec.scenarios is set

TestTrigger generation

  • Create TestTrigger when spec.trigger.enabled: true, watching the Deployment derived from agentRef
  • Delete/skip TestTrigger when disabled or omitted

Config parameter mapping

  • Resolve agentRef to agent URL (Agent CR status or service DNS)
  • Pass llmAsAJudgeModel into experiment JSON
  • Inject OTEL endpoint from cluster ConfigMap (otel-config)

Ownership & garbage collection

  • Set ownerReferences on all generated resources (ConfigMap, TestWorkflow, TestTrigger)
  • Verify cascade deletion works correctly

Status reporting

  • Update Ready and WorkflowReady conditions on the Experiment CR after each reconciliation
  • Track generated resource names in status.generatedResources

Validation webhook (stretch)

  • Reject CRs with both dataset and scenarios set
  • Validate agentRef references an existing Agent CR

Tests

  • Unit tests for reconciliation logic (envtest or mocked client)
  • Unit tests for trigger creation/deletion logic and status updates
  • CRD validation tests (invalid thresholds, missing required fields)
  • E2E test: create Experiment CR → verify ConfigMap, TestWorkflow, TestTrigger generated → delete CR → verify cascade deletion
  • golangci-lint passing

Acceptance Criteria

  • Creating an Experiment CR generates ConfigMap + TestWorkflow + (optional) TestTrigger
  • Updating an Experiment CR updates generated resources accordingly
  • Deleting an Experiment CR cascades deletion via ownerReferences
  • TestWorkflow correctly chains phase templates based on data source mode
  • Agent URL resolved from agentRef
  • Experiment JSON correctly serialized into ConfigMap from inline scenarios
  • TestTrigger created/deleted based on spec.trigger.enabled
  • Status conditions and generatedResources updated after reconciliation
  • Unit tests and E2E test passing
  • golangci-lint passing

References

Comments on the Issue (you are @copilot in this section)

🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.

Copilot AI and others added 2 commits March 17, 2026 12:02
@fmallmann
Implement Experiment reconciler with ConfigMap, TestWorkflow, TestTri…
4843de3 
…gger, status reporting, and tests

Co-authored-by: fmallmann <30110193+fmallmann@users.noreply.github.com>
@fmallmann
Final verification and documentation complete
23e6701 
Co-authored-by: fmallmann <30110193+fmallmann@users.noreply.github.com>
Copilot AI changed the title [WIP] [agentic-layer/testbench#27] Implement reconciler logic and tests Operator: full reconciler implementation with status reporting and tests Mar 17, 2026
Copilot AI requested a review from fmallmann March 17, 2026 12:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fmallmann fmallmann Awaiting requested review from fmallmann

Assignees

@fmallmann fmallmann

Copilot code review Copilot

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Operator: Reconciler implementation, status reporting, and tests

2 participants

@fmallmann